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Abstract 

We cryptanalyse a matrix-based key transport protocol due to Baum- 
slag, Camps, Fine, Rosenberger and Xu from 2006. We also cryptanal- 
yse two recently proposed matrix-based key agreement protocols, due to 
Habeeb, Kahrobaei and Shpilrain, and due to Romanczuk and Ustimenko. 



1 Introduction 

Regular proposals are made to employ groups in cryptography; see for example 
the survey article by Blackburn et al [2j or the book by Myasnikov et al [5J. 
In particular, matrix groups are often considered because matrices are easy 
to represent and manipulate. However, such proposals generally have a poor 
reputation: we are unaware of any fully specified proposals that are widely 
regarded as secure. 

In this paper we cryptanalyse a matrix-based key transport protocol due 
to Baumslag, Camps, Fine, Rosenberger and Xu F, which we refer to as the 
BCFRX scheme. In fact, their proposal is more general and they suggest several 
platform groups; we consider their only matrix group proposal. We cryptanalyse 
this scheme in a very strong sense. We show that for practical parameter sizes 
a passive adversary can feasibly recover the session key after observing just one 
run of the protocol. We find an even more efficient attack if two or more runs 
of the protocol are observed. Our techniques reduce the problem of breaking 
the scheme to a sequence of feasible Grobner basis computations. This work 
constitutes Section 2. 
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We also cryptanalyse two recently proposed matrix-based key agreement 
protocols, due to Habeeb, Kahrobaei and Shpilrain (HKS) [4], and due to Ro- 
manczuk and Ustimenko (RU) [7]. These schemes both fail due to straightfor- 
ward linearisation attacks. This work constitutes Sections 3 and 4. 



We begin by describing the BCFRX scheme. The protocol assumes that Alice 
and Bob a priori share some secret information, namely their long-term secret 
key. The goal of the protocol is for Alice and Bob to establish a session key for 
subsequent cryptographic use. To achieve this, Bob chooses the session key and 
sends it to Alice in three passes, as follows. 

Let Q be a finitely presented group. Let A and B be two commuting sub- 
groups of Q (so AB = BA for all A € A and B G B). The group Q is made 
public and the subgroups A and B form Alice and Bob's long-term secret key. 
Then: 

• Bob chooses a session key K £ Q and elements B, B' e B. He sends 
C := BKB' to Alice. 

• Alice picks elements A, A' 6 A and sends D := AC A' = ABKB'A' to 
Bob. 

• Since A and B commute, we have that ABKB'A' = BAKA'B'. Bob 
sends E := B^DB'^ 1 = AKA! to Alice. 

• Alice computes K = A~ 1 EA'~ 1 . 

We can think of this protocol as Shamir's three-pass (or no-key) protocol [H 
Protocol 12.22, Page 500], with the operation of multiplying on the left and 
right by a group element replacing the exponentiation operation. 

There was no detailed discussion of security in pQ, but we need to specify 
a security model and what it means to break the protocol, in order to crypt- 
analyse it. We will consider the weakest possible notion of security: the passive 
adversary model. So we will regard the protocol as broken if we can construct 
an adversary that can feasibly compute the session key, after eavesdropping on 
one or more runs of the protocol; this adversary must perform well for practical 
parameter sizes. 

Baumslag et al. Q] suggested several abstract platform groups to serve for Q. 
But in this paper we concentrate on their only matrix group proposal: Q = 
SL4(Z), the group of invertible 4x4 matrices of determinant 1 over the integers. 
It was proposed that the commuting subgroups A and B should be constructed 
as follows. Writing I2 for the 2x2 identity matrix, define the subgroups U and 



2 The BCFRX Scheme 
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Let M G SL 4 (Z) be a secret matrix known to both Alice and Bob. Then we 
define 

A = M- X UM and B = M^CM. (2) 

We may thus view the long-term secret key as the matrix M. 

As described the proposal is not yet fully specified, since it remains to specify 
how the long-term secret key M is chosen, and how the protocol chooses elements 
from A and B at various points. It was stated in Baumslag et al. Q] that elements 
are picked randomly from A and B, and we presume that the matrix M is picked 
in a similar fashion from Q — SL 4 (Z). But since the group Q and its subgroups 
A,B are infinite, the meaning of the word random is unclear in this context. 
Any practical cryptanalysis will depend on the details of how these random 
choices are made; however the cryptanalysis we give below will work for any 
efficient method for making these random choices that we can think of. 

In any fully specified implementation of the protocol, there exists an integer 
A such that the entries of all matrices generated in the protocol lie in the interval 
(— A/2, A/2). Since the standard way to represent a 4 x 4 integer matrix of this 
form uses approximately 16 log 2 A bits, it is natural to think of log 2 A as the 
security parameter of the scheme. 

A cryptanalysis 

Our cryptanalysis proceeds in three stages. In Stage 1, we argue that integer 
computations may be replaced by computations modulo p for various small 
primes p. In Stage 2 we show that knowledge of a matrix iV of a restricted form 
allows a passive adversary to compute any session key transmitted under the 
scheme. Finally, in Stage 3, we show that this matrix N may be computed in 
practice. None of these stages is rigorous (though Stage 2 may be made so) , but 
the stages all work well in practice. 

Stage 1: Working modulo p 

Suppose an adversary wishes to discover a session key K. Since the entries of K 
lie in the interval between —A/2 and A/2, it is enough to find K mod n for any 
n > A. Indeed, this is how we approach our cryptanalysis. We will show (see 
Stages 2 and 3 below) that in practice we may efficiently compute K mod pi for 
small primes p% of our choice. (We are thinking of p, as a prime of between 80 
and 300 bits in length: in some sense quite large, but in general smaller than A.) 
We run this computation for several different primes pi until YiPi > A. Setting 
n = YiPii we can then appeal to the Chinese remainder theorem to calculate 
K mod n = K . 

We write this more precisely as follows. Let T be a fully specified version of 
the BCFRX protocol, with SL 4 (Z) as a platform. For a prime p, let Z p be the 
integers modulo p. Let T p be the BCFRX protocol under the platform group 
Q = SL 4 (Z p ), defined as follows. We identify the subgroups U and C defined 
by with their images in SL 4 (Z p ). Let the subgroups A and B be chosen to 
be of the form (|2]) for some matrix M € Q chosen uniformly at random. Let 
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the protocol pick all elements from A and B uniformly and independently at 
random. This makes sense since Q is finite. We use T p to model the protocol 
T taken modulo p. This model is not quite accurate: for example, it is almost 
certain that when M G SL4(Z) is chosen according to the method specified in 
T, the distribution of M modp will not be quite uniform in SL^Zp). But for 
all ways we can think of in which T can be specified, the protocol T p is a good 
model for T taken modulo p (in the sense that an adversary that succeeds in 
practice to recover the session key generated by T p will also succeed in practice 
to recover K mod p when presented with the matrices from a run of the protocol 
T) . Note that an adversary has great freedom in choosing p, which makes the 
reduction to T p difficult to design against. The fact (see below) that the session 
key for T p can be feasibly computed in practice shows that T is insecure. 

Stage 2: Restricting the long-term key 

We consider the protocol T p over SL 4 (Z p ) defined above. From now on, let us 
write an arbitrary 4x4 matrix Z in block form as Z = ( f ^ f ^ ) , for the obvious 
2x2 submatrices Zij of Z. 

The following lemma shows that there are many equivalent long-term keys 
for the protocol T p . 

Lemma 2.1. Let M G SL^Zp) be the long-term key shared by Alice and Bob, 
and define subgroups A and B by A = M~ X UM and B = M _1 CM '. Let N G 
GL 4 (Z p ) be any matrix such that N'^UN = A and N^CN = B. If N is 
known, then any session key can be efficiently computed by a passive adversary. 

Proof. An adversary is presented with matrices C, D and E that are transmitted 
as part of the protocol. We have that C = BKB', D = ABKB'A' and E = 
AKA' for some unknown matrices A, A' £ A and B, B' G B. Suppose that 
the adversary is also able to obtain a matrix N satisfying the conditions of the 
lemma. Since A,A'eA we may write A = N~ 1 RN and A' = N~ 1 R'N for 
some unknown matrices R,R' G U. Similarly we may write B — N~ l SN and 
B' = N^S'N for some unknown matrices S, S' G C. 



Define an (unknown) matrix K' by K' = NKN 1 . Define matrices C, D', 
E'hy 



c 

D' 
E' 



NCN- 1 = SK'S', 
NDN- 1 = RSK'S'R' and 
NEN- 1 = RKR'. 



Note that the adversary can compute C , D' and E'. 
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Using the fact that S, S' £ C and R, R' £ U, we may write 



C" 



S22K' 21 S22K 22 S'22 



and 



D' = ( ^ n -^ii^'n ^11-^12^22 

V $22^21-^11 S22K22S22 

F' — { RnK'uR'u RuK'12 
\ K 21 R' n K' 22 

Clearly K' n is known to the adversary, since K' n — C' n . Moreover, K 22 is 
known since K 22 — E' 22 . 

To compute K' 12 , find any matrix X such that XD' 12 = C 12 (note there may 
be more than one such X if K' l2 is noninvertible). This implies XR\\K' 12 — 
K' 12 , since S' 22 is invertible. Thus an adversary can compute XE' l2 = K' l2 . 
Similarly, to compute K' 2l find any matrix Y such that D' 21 Y = C 21 . This 
implies K 21 R' n Y = K' 2l and an adversary can compute E' 21 Y — K' 21 . 

Once K' is known, the session key K may be recovered since K = N^ 1 K'N . 

□ 

Let Mat2(Z p ) be the set of 2 x 2 matrices over Z p . Let I C Mat2(Z p ) be 
defined by 

' '\ 0\ fl 0\ (0 0^ 

We say that JV £ GL 4 (Z p ) is of restricted form if iVn, 7V 2 2 € X. 

Lemma 2.2. For any long-term key M used in the protocol T p , there is a matrix 
N of restricted form satisfying the conditions of Lemma \2.1l Moreover, for an 
overwhelming proportion of long-term keys M , we may impose the condition 
that Nn = N22 = I2, where 1% is the 2x2 identity matrix. 

Proof. Let / : Mat 2 (Z p ) -)• GL 2 (Z p ) be a function such that f(X)X £ 2 for 
all X £ Mat2(Z p ). Such a function / certainly exists: it can be derived from a 
standard row reduction algorithm. 
Define 

The definition of H means that N±i, N22 £ 2, and so N is of restricted form. 
Also, any matrix 

/GL 2 (Z P ) 
fc ^ GL 2 (Z P ) 
has the property that H^UH = U and H~ l CH = C. So 

N^UN = M~ 1 H~ 1 UHM = M~ X UM = A 

and similarly B — N^^^CN. So the main statement of the lemma is proved. To 
see why the last statement of the lemma holds, note that for an overwhelming 
proportion of long-term keys M we have that Mn and M22 are invertible. The 
function / maps any invertible matrix to its inverse, and so Nn = N22 = I2 in 
this case. □ 
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Stage 3: Computing the matrix N 

We may compute an equivalent long-term key N of restricted form as follows. 
After eavesdropping on a run of the protocol, we know the matrices C,D, and 
E. We also know a matrix N of restricted form must satisfy the equations 

NDN- 1 = RNCN^R', (3) 
NDN- 1 = SNEN^S', (4) 
NN' 1 = h, (5) 

for unknown matrices R,R' eW and S,S' £ C. Since N is of restricted form we 
have Nn,N22 £ !• There are thus only 9 possible combinations for N\\ and N22, 
so we may perform a trivial exhaustive search to find the right combination. (In 
practice we would first try Nu = N22 = h, since this holds with overwhelming 
probability.) We assign variables xi, ...,xg for the remaining unknown entries 
of N, and xg, . . . , X24 for the unknown entries of N . 
Expanding (|3J) and (J3J, we find 

(NDN- X ) 22 = (NCN~ 1 ) 22 , (NDN- 1 ),, = (NEN~ 1 ) n . 

This gives us 4 + 4 = 8 quadratic equations in the Xi,i = 1, . . . , 24. Adding the 
16 quadratic equations from ([5]), we have a system of 24 quadratic equations 
in 24 unknowns and expect a Grdbner basis calculation to reveal N. If we 
eavesdrop on a second run of the protocol, we learn 8 new equations (from ([3]) 
and and expect to compute N even more efficiently. 

Experimental results 

Over 1,000 trials using Magma [3] Version 2.16-11 on a Intel Core 2 Duo 1.86GHz 
desktop, it took roughly 12 seconds to compute each (lex ordered) Grobner basis 
for a random 300-bit prime. In all our experiments, twenty three of the basis 
elements had the form x\ + fi(x2^) for i = 1, . . . , 23, where is a polynomial 
of degree 5. The final basis element was a degree 6 polynomial in X24. Thus in 
all our cases we had a maximum of six possibilities for a matrix N of restricted 
form satisfying Lemma |2. II 

If we eavesdrop on a second run of the protocol, we can add 8 new equations 
(or just one of the 8 new equations) to our system. A Grobner basis calculation 
then reveals a unique value for N. 

3 The HKS Scheme 

Next we turn our attention to a key agreement protocol proposed by Habeeb, 
Kahrobaei and Shpilrain 4 . Our description of the scheme is somewhat sim- 
plified from [3]. 

Let A be a group and let B be an abelian group. Let Aut(B) denote the 
automorphism group of B, and let A, B, Aut(B), a £ A, b e B, n e N be public. 
Then 
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Alice picks an embedding ip : A — >■ Aut(B) and sends 
x = iP{a){b)^{a 2 ){b) . . . 4>{a n - l ){b) to Bob. 

Bob picks an embedding (f> : A — >■ Aut(S) and sends 
y = 0(a)(6)0(a 2 )(6) . . . ^(a™- 1 )^) to Alice. 

• Alice computes 

n—1 n—1 n—1 

i—1 i—1 j—1 

• Bob computes 

n— 1 n— 1 n—1 

k B = n 0( o< )(«) - n n ^w^w- 

We require that Alice and Bob pick ip and (/> so that they commute. If this is 
done, Alice and Bob have computed a common shared key k = k A = ks- 

The proposal 4. suggests to take A to be a p-group (a group of order p l for 
some I G N and prime p) and B to be an elementary abelian p-group of order 
p m . Thus B may be viewed as an m-dimensional vector space over F p , and 
so Aut(i?) = GL m (F p ). With this choice of platform groups, we can view the 
protocol as follows. 

Define f(x) — x + x 2 + ■ ■ ■ + a;' 1-1 . Let b be an m-dimensional column vector 
over F p . Alice and Bob choose private m x m matrices J and K respectively, 
using some method so that /(J) and f(K) commute. In general, and a little 
more formally, J = Ma(ta) and K = Mb(tb) where Ma and Mb are public 
algorithms which take as input random sequences of coin tosses r A and tb re- 
spectively (in addition to the public parameters of the scheme). The algorithms 
must have the property that the matrices /(M^rvt)) and /(Ms(rf,)) commute 
for all input sequences ta and rs respectively. The paper [3] suggests some 
candidates for Ma and Mb, but we do not make use of the details of these 
algorithms in this cryptanalysis. 

Alice transmits the column vector wa = f(J)b to Bob. Bob transmits the 
column vector wb = f{K)b to Alice. The common key k is the column vector 
defined by 

k = f(J)f(K)v = f(J)w B = f(K)w A , 
the last equality following since f(J) and f{K) commute. 



A cryptanalysis 

Suppose an adversary Eve receives wa,wb and the public parameters of the 
scheme. 

Let X be any matrix such that Xb — wa, and X commutes with f(L) for all 
matrices L that can possibly be generated by Bob. Such a matrix exists since 
X = f(J) satisfies these conditions. 
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Note that the conditions on X are linear conditions on the unknown entries of 
X. This is clear for the condition that Xb = wa- The commutator condition can 
be expressed as Xf(L) = f(L)X, for matrices L output by the algorithm Mb- 
To compute the commutator condition on X , Eve can run Mb on some random 
inputs te to find suitable matrices f(L) and impose the necessary conditions 
Xf(L) = f(L)X on X. Since these conditions are linear, the number of random 
inputs te that is required before these necessary conditions become sufficient 
to imply the commutator condition (at least for an overwhelming proportion of 
runs of the protocol) is very small. 

Since all the conditions on X are linear and easy to find, a suitable matrix 
X can be computed efficiently. 

We claim that k = Xwb- To see this, observe that 

Xw B = Xf(K)b = f(K)Xb = f(K)w A = f(K)f(J)b = f(J)f(K)b = k. 

This means that the adversary can generate the shared key, and the scheme is 
broken. 

4 The RU Scheme 

We now cryptanalyse a recent key agreement protocol proposed by Romanczuk 
and Ustimenko [7j. The protocol works as follows. 

Let GL„(F g ) denote the group of invertible n x n matrices over a finite field 
W q of order q, and let V q [x, y] denote the polynomial ring over ¥ q in two variables 
x and y. Let C,D £ GL n (F g ) be two commuting matrices and let d £ ¥ q . The 
matrices C, D and the vector d are made public. 

To agree on a shared key, Alice picks a polynomial fA{x,y) £ F g [x,?/] and 
sends wa = fA(C,D)d to Bob. Likewise Bob picks a polynomial y) £ 

¥ q [x,y] and sends wb = /b(C, D)d to Alice. Alice computes fc^ = Ja(C,D)wb, 
Bob computes ks = /s(C, D)wa- Since C and D commute, the same is true 
for Ja(C, D) and /s(C, D), and so their shared key is the vector k :— kA = ks- 

It was not fully specified how the matrices C, D and the polynomials /a, Jb 
are generated. However, the following cryptanalysis applies to any method of 
generation. 

A cryptanalysis 

Suppose a passive adversary Eve receives wa, and the public quantities C, D 
and d. Let X be any matrix such that 

XC = CX, XD = DX, Xd = wa- 

Note that such a matrix exists, since X = Ja{C,D) satisfies these conditions. 
Since the conditions on X are all linear, such a matrix is easily found. Eve can 
then compute the key as: 

Xw B = Xf B {C,D)d = f B (C,D)Xd - f B (C,D)w A = k. 
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